Decentralized study of COVID Vaccine Antibody Response (STOPCoV): Results of a participant satisfaction survey

The Covid-19 pandemic required many clinical trials to adopt a decentralized framework to continue research activities during lock down restrictions. The STOPCoV study was designed to assess the safety and efficacy of Covid-19 vaccines in those aged 70 and above compared to those aged 30–50 years of age. In this sub-study we aimed to determine participant satisfaction for the decentralized processes, accessing the study website and collecting and submitting study specimens. The satisfaction survey was based on a Likert scale developed by a team of three investigators. Overall, there were 42 questions for respondents to answer. The invitation to participate with a link to the survey was emailed to 1253 active participants near the mid-way point of the main STOPCoV trial (April 2022). The results were collated and answers were compared between the two age cohorts. Overall, 70% (83% older, 54% younger cohort, no difference by sex) responded to the survey. The overall feedback was positive with over 90% of respondents answering that the website was easy to use. Despite the age gap, both the older cohort and younger cohort reported ease of performing study activities through a personal electronic device. Only 30% of the participants had previously participated in a clinical trial, however over 90% agreed that they would be willing to participate in future clinical research. Some difficulties were noted in refreshing the browser whenever updates to the website were made. The feedback attained will be used to improve current processes and procedures of the STOPCoV trial as well as share learning experiences to inform future fully decentralized research studies.


Introduction
Beginning in January 2020, the COVID-19 pandemic caused unprecedented illness and death, economic crisis and severe disruption to normal daily life globally and in Canada [1,2]. The elderly and those with comorbidity were amongst the most vulnerable [3]. Preventative vaccines were made available at exceptional speed and the initial protective rate of 90-94% for the 2-dose messenger RNA vaccines was far better than could have been anticipated [4,5]. The clinical trials, although adequately powered and of gender parity, primarily enrolled white persons < 70 years of age with minimal comorbidity not reflective of the Canadian population. Questions remained including the protective antibody level, the extent and durability of the vaccine serologic response, long term efficacy, the risk of breakthrough infection and the impact on any emerging new variant strains [6][7][8][9][10][11][12][13][14]. Further, the National Advisory Committee on Immunization in Canada recommendations on longer delays between vaccine doses [15] than used in the clinical trials in attempt to partially immunize more of the population added further uncertainty [9,[16][17][18][19].
Faced with these questions, we felt an urgent unmet need to determine the long term antibody response data to COVID-19 vaccines outside of the clinical trial setting to inform the potential need for boosters particularly among the elderly. Similar work was being conducted in long term care facilities where the death rate from COVID-19 infection was the highest [20]. However, the results of these studies would not necessarily translate to an older ambulatory elderly population. The latter type of data would be necessary to inform our health officials to consider the timing for lifting of restrictions and to inform the future needs for masking and other public health measures.
Traditionally clinic trials are conducted with onsite based approaches as this allows researchers to closely monitor each study participant in a controlled setting [21]. The Covid-19 pandemic highlighted the limitations of centralized clinical trials. Public health restrictions forced hospital and doctors office-based study coordination centers to temporarily close their sites for any non-essential in-person research visits, disrupting enrollment and follow up research activities [22]. In response, many centralized clinical trials adapted their research methods to enable virtual and electronic data collection.
We will transfer relevant anonymized study data as available to the CITF as a part of these standard data sharing requirements. This is submitted together with a data dictionary defining each field in the set. External researchers will be able to submit a request to the CITF to receive access to all CITF data through their data access committee. The CITF will employ a rigorous checklist to ensure that these external requests follow all necessary ethical and privacy protocols. The data provided to the CITF will be stored on the CITF Database. The data on the CITF Database will be held under the custodianship of McGill University or one of its collaborators and be shared via the cloud, both nationally and internationally. Data in the CITF Database can be used by researchers across Canada and in other countries following Data Access Committee (DAC) approval. These transfers will also be made in compliance with Canadian law and research ethics. A DAC will be responsible for reviewing applications for access to the data and for approving applications that respect the privacy and access policies of the CITF. The DAC will require that researchers confirm that their intended research activities have received necessary ethics approvals. The data may also be shared with other COVID-19 research databases that follow similar protections and procedures as the CITF Database. Further the main study protocol, statistical analysis plan, informed consent form and full protocol are available on the study website www.stopcov.ca.
Given these limitations in conducting research, we designed the STOPCoV study (Safety and Efficacy of Preventative COVID Vaccines) in the midst of the global Covid-19 pandemic in order to assess the impact of age (� 70 years vs. 30-50 years) on the antibody response and the safety of the approved vaccines over time (www.stopcov.ca) [23]. This clinical trial needed to be decentralized to meet public health restrictions, and have a flexible design and regulatory approval in order to adjust to changing vaccine recommendations from public health officials in Ontario [3]. The Data Aggregation, Translation and Architecture (DATA) and Healthcare Human Factors (HHF) teams at the University Health Network (UHN) collaborated with the study team to develop an online digital platform to enable decentralized study procedures such as recruitment, informed e-consent (with videos and interspersed questions to re-enforce comprehension), baseline and follow-up questionnaires, symptom diaries, monthly check-ins, video instructions, and email reminders for study activities. Other study procedures such as dried blood spots were completed via mail. With the Omicron wave of the epidemic, it became clear that breakthrough infection could occur despite vaccination. In response, we added a sub-study to our protocol to estimate the rate of, and factors associated with, breakthrough infections and the predictive value of the rapid antigen tests in diagnosing infection in a vaccinated population (4). Again, we needed to adapt our digital platform to enable consent, training of our participants and data capture for this sub-study.
This was a new research method for us and our institution. We had to rapidly develop our platform and make frequent changes due to the rapid changing recommendations of our government with respect to vaccine doses, brands and intervals. We anticipated that many of our participants had never been previously involved in research and may not have had much experience with digital platforms. Consequently, we designed this survey to determine the satisfaction of our participants with our protocol, tools and procedures. We hypothesized that age could have an impact on satisfaction given the need for some digital proficiency and confidence to participate. Our aim was to identify areas to improve the process and procedures and share our experience to inform future fully decentralized research studies.

Methods
Full methods for the main STOPCoV study [23] and the rapid antigen test sub-study [24] are available but are described briefly below.

Design
A longitudinal cohort study planned to follow participants with two COVID-19 vaccine doses for 48 weeks. Trial registration: Clinicaltrials.gov. NCT05208983. Participants were eligible if they were between 30-50 years or �70 years and received their vaccine through an Ontario Distribution center. We purposely planned for 70% enrollment in the older cohort.

Recruitment
A data sharing agreement with the Ontario Ministry of Health enabled us to send study information emails to persons receiving the COVID-19 vaccine at an Ontario distribution center who consented to contact for research. A similar email was sent to Ontario Canadian Association of Retired Persons members (www.carp.ca). Participants could enrol through the website prior to the first or second vaccine dose that had to be administered in Ontario. A total of 1286 adults (911 older and 375 younger) self-recruited between May 17-July 31, 2021. Five participants did not meet eligibility criteria and at the time of initiation of the rapid antigen substudy, 20 had withdrawn consent, leaving 1261 (98%) continuing in the cohort. These active participants were sent an email describing the sub-study in mid-January 2022. 806 participants e-consented and 727 (90%) completed > 1 rapid antigen test (RAT) January 28-March 29, 2022.
Electronic consent of the initial and substudy including the request to share core data elements with the Canadian Immunity Task Force was completed on the study website. The study and electronic consent process were approved by the University Health Network (UHN) Ethics Review Committee. Consented participants used the study website with their personal identification (ID) number and password as a portal for communication with study staff, data collection and results reporting. A schedule for required activities and email reminders were provided.

Questionnaires
Self-administered electronic questionnaires collected baseline demographic and health data. Racial background was self reported. Electronic diaries collected data on vaccine date and brand and local and systemic adverse events for 7-days after the first three vaccine doses. Monthly check-in questionnaires capture persistent vaccine related adverse events, booster doses and new COVID-19 diagnoses.

Dried Blood Spot (DBS) Specimens
Samples were requested +/-7 days of initial vaccine, three weeks (+/-1 week) after the first vaccine dose, two weeks (+/-1 week) after the second vaccine dose and then every 12 weeks (+/-3 weeks). If the dose interval exceeded 28 days, an additional sample was collected prior to the second vaccine dose. Additional DBS were requested 3-4 weeks after vaccine boosters. DBS collected on Whatman 903 cards using a lancet for finger-prick.

Satisfaction survey
The participant feedback survey was modelled after the 'Study Participant Feedback Questionnaire Toolkit' created by TransCelerate Biopharma [25] and included questions related to each phase of the STOPCoV study and the rapid antigen test sub-study, including recruitment, consent process, study procedure, digital platform and interaction with the study team. The questions were designed to be answered on a 5-point Likert scale. After an initial set of questions was developed, 3 members of our research team met in person to determine by consensus which questions should be included and to ensure they were comprehensive and clear. A final set of 42 questions that could be answered in under 15 minutes was formed and tested with three non-study participants in the same age range as our participants to obtain feedback on clarity and ease of use. REDCap, an external secure web platform, was used to distribute the survey via email to active participants and collect responses. The survey was reviewed and approved by the University Health Network Research Ethics Board.
Demographic and clinical characteristics of participants who responded to the survey were compared to those who did not using χ 2 -tests for categorical variables and Wilcoxon Rank Sum tests for continuous variables. Only 3 individuals reported their gender as non-binary, they have been included in the female category for analysis. Individual survey item responses were summarized by age cohort using counts and percentages, and comparisons of the items were made between age cohorts using χ 2 -tests for trend. For the comparisons, non-responses were included as separate categories, and for the Likert-type questions with five possible categories, we formed two categories by combining the top two levels and combining the bottom three levels and non-responses (e.g. "Strongly Agree" or "Agree" were compared to "Neither Agree nor Disagree", "Disagree", "Strongly Disagree", or "Not Answered"); an α-level of 0.05 was used.

Results
The survey link with an introduction to the purpose of this sub-study was sent by email to 1253 active participants near the mid-way point of the main STOPCoV trial (April 2022). They were informed that they would not be identified by name and were not required to answer all questions. Of these, 726 had also participated in the rapid antigen test sub-study. In total 883 (70%) responded to the survey including 701 (83%) from the older and 182 (54%) from the younger cohort, with no difference by sex. We sent one reminder to those who did not complete the survey after 8 days. We closed the survey 14 days after the first email inviting participation. Table 1 compares the self-reported demographics of those who responded vs those who did not respond to the survey. While all of the participants who did the survey also completed the baseline questionnaire for the main study only 80.3% of the people who did not do the survey had baseline data available (p<0.0001). For the participants who did have baseline data: those who did the survey were more likely to be from the older cohort (79% vs. 49% p<0.0001), white (90% vs. 82%, p = 0.0015), previous smokers (40% vs. 31% p<0.0001), and have underlying cardiovascular disease (39% vs. 28%, p = 0.0006) or a cancer diagnosis (17% vs. 9%, p = 0.00052) compared to those who did not complete the survey. Gender, other comorbidities, were not different by survey participation. Table 2 shows the responses to items regarding clinical research experience. Prior to the STOPCoV study, only 33% of both male and female participants had participated in a clinical research study more in the older cohort. However, 47% of the participants agreed that the Covid-19 pandemic changed their awareness of clinical research studies. 95% of participants were satisfied to very satisfied with respect to their participation in the STOPCoV clinical trial and 90% indicated they would participate in research again. Responses to these questions did not vary by age category or sex.
Recruitment strategies, reasons for participating in the main STOPCoV study, and questions about the e-consent process are shown in Table 3. Recruitment in the older cohort was mainly from response to an email sent from the study team (40%), or a targeted email from a seniors magazine mailing list (ZOOMER/CARP) (14%) or referral from a friend or relative (13%). The younger cohort were most likely to have responded to an email from the study team (36%) or from a referral for a friend or relative (28%). The younger cohort (especially the females) were more likely to have enrolled based on learning about the study through social media like Facebook or twitter 19%) compared to 2% of the older cohort.
The leading reasons participants decided to enroll were to help others learn more about the antibody response to the COVID vaccine (76% overall, 90% younger, 72% older cohort, p < .001)), to learn about their own personal antibody response (68%, overall, 84% younger, 65% older cohort, p < .001) especially the females (72% vs 61% males, p = .001), to contribute to knowledge about the safety of COVID vaccines (73%) as well as the convenience of taking part in a clinical trial from home (37%). The latter two factors were more commonly reported by the females.
Over 90% of participants (no difference by age group or sex) responded that they understood what study procedures were expected of them before consenting to participate. Over 60% of participants (67% of the older cohort, 60% younger cohort, p.005); no difference by sex watched the consent video and 65% of the older and 58% of the younger cohort, p.085) found the explanation of the consent process helpful.
All reporting for the main study was done through the STOPCoV website; satisfaction questions about the study website and the schedule of the study are shown in Table 4. Overall, 90% of participants in both sex and age categories agreed/strongly agreed the study website was easy to use. Both age cohorts agreed/strongly agreed that it was easy to log in and participate in the study using their electronic device. Participants reported that the monthly questionnaires were easy to understand and they were able to navigate through the website with ease to record results and symptoms in their profile. Over 80% of participants in both age and sex cohorts The strongly agreed that the email reminders that were sent when monthly questionnaires and DBS were due were very helpful. Whenever changes were made to the design of the study website to collect new information, the site needed to be refreshed/cache cleared; 12% of participants reported that they had difficulty doing this more commonly in the younger cohort 17% vs 11% of the older cohort, p = .027) and in the women 4.2% compared to the men 3.5%, p = .012. Over 80% of participants (no difference by age or sex category) agreed that compared to when the study started, the overall commitment required was close to what they had expected. Table 5 shows responses to questions regarding dried blood spot collection and rapid antigen tests. Initially, there was some confusion about how to use the lancets to collect the dry blood spot (DBS) and many participants called in for instruction. Although not available at the study start, 70% of the study participants subsequently watched the video explaining the dried blood spot collection procedure and found the instructional video helpful. While 78% of respondents answered there was little to no difficulty using the lancets and collecting their DBS (the men reported slightly more problems, p = .027), 39% did report difficulty getting sufficient blood to put on the DBS collection card at least some of the time, but this did not differ by age or sex category. Only a small per cent reported unacceptable discomfort with the blood prick. Over 90% in both age and sex cohorts found the frequency of DBS collection was acceptable. Over 75% of participants felt that their antibody results were presented in a way that was easy to understand (no sex difference) but this was less common in the older 78% than the younger cohort 85%, p = .036).
An instructional video was available at the time of recruitment to the Rapid Antigen Test (RAT) sub-study. 41% of the participants (43% older, 36% younger cohort, p < .001) reported watching this video with 41% finding it helpful, and 69% older and 76% younger, p = .097) participants felt confident interpreting the results of the RAT with no difference by sex.
Interactions with study team are described in Table 6. A dedicated study email and phone line for contacting the study team was available to address concerns and provide assistance. The study team was contacted frequently by phone 38% and by email 66%. Overall, the older cohort was more likely to contact the study team by phone 44% vs 18% of the younger cohort, p = .001. In contrast the younger cohort were more likely to contact the study team by email (74% vs 64%, p = .01) Females were more likely to use email 69% than men 60%, p < .01. Participants in both age cohorts (especially the females) answered that the study staff was both easy to reach (80%) and treated them with respect (90%). 79% of both age cohorts especially the females (85% vs 74% men, p < .001) were satisfied with the answers to their questions.

Discussion
Overall, the responses to the feedback survey were positive and 95% of participants in both age and sex cohorts were satisfied to very satisfied with respect to their participation in the STOP-CoV clinical trial. This is further exemplified by the high proportion of respondents to this satisfaction survey. The older cohort and those with underlying cardiovascular disease or cancer were more likely to respond to the survey which is consistent with their overall study participation and may reflect concern for their own health. Main reasons for participation in the STOPCoV study were to learn about their own personal antibody response (68%) as well as help others learn more about the antibody response to the COVID vaccine (76%). This was an important observation as many research ethics boards/researchers do not allow research results to be given directly to participants as their significance may be unclear and misinterpreted [26,27]. However, this was an evolving pandemic and providing research participants with their antibody levels at multiple time points helped  with retention and to maintain participant engagement throughout the study. Participants were provided with a graph showing the results of individuals in their age cohort in relation to their individual values. A statement in the consent form and on the results page of the website reinforced that the results were research, the significance was unknown, and the team was available for questions. Although most reported in the survey that they were able to interpret their results many did contact the study team with questions of clarification.
Recruitment to the main STOPCoV study was rapid; 39.4% responded to an email sent by the study team based on their consent to be contacted for research at the time of their vaccination. The use of a senior's magazine mailing list (ZOOMER/CARP) to engage the older participants was successful with 11% reporting this as their primary contact. Another 5% responded to recruitment through social media (Facebook/Twitter) but this was much more successful with the younger and female participants where 19% reported this as the means of learning about the study. These responses highlight that recruitment strategies need to reflect the population of interest.
With the pandemic, electronic consents are increasingly being utilized, but there is controversy about whether they will be acceptable and understandable. [28] The majority of our participants (65%) especially the older group responded that they viewed the video posted on the STOPCoV website explaining the consent process and 64% found it helpful.
We had concerns about how easy it would be for our participants to use the study website to enter their data and that the older cohort might have trouble manipulating their portal. Furthermore, due to rapidly evolving vaccine recommendations and associated study procedures, the digital platform was not complete when the study was launched and every change required a page refresh and/or clearing cache which many of our participants were not familiar with. Despite these concerns, most respondents in both age and sex cohorts reported the study site easy to use suggesting that our participants were digitally proficient. We had anticipated that more of the older cohort would report caching issues but this was opposite to what was reported in the survey. This may have been a result of misinterpretation of the statement *Chi-squared test for trend (exact test used when expected cell counts were low) †Chi-squared test for dichotomized response categories "Strongly Agree" or "Agree" vs. others (including "Not Answered") "All of the time" or "Most of the time" vs. others (including "Not Answered") "Too much" or "A little too much" vs. others (including "Not Answered") https://doi.org/10.1371/journal.pdig.0000242.t005 which was a double negative or because the older cohort were more likely to contact the study team for instructions on how to do this. The study procedures involved the need to use lancets to collect DBS. In our survey, 37% of the younger and 43% of the older cohort (40% men, 43% women) described difficulty with this procedure at least a little of the time. In order to improve this process, the team was available by email or phone to address concerns. There were many such contacts at the beginning of the study, which diminished with time. In response, the study team developed a short video to demonstrate how to use the lancets. This was not available at the time of the first sampling but in the survey 70% participants responded that they found the video helpful. Before it was available online, we sent an email with a picture of the lancet and its proper usage. We learned from this experience and ensured the instructional video for the rapid antigen test was readily available when the sub-study was initiated. There were fewer calls expressing difficulty with this procedure and the majority found the antigen tests easy to do and record.

PLOS DIGITAL HEALTH
In the survey 39% of both age and sex cohorts reported difficulty in getting sufficient blood to put on the DBS collection card. With the next shipment of DBS collection kits, the study team included two additional lancets of a different needle size. However, this created some confusion as their appearance was different and resulted in an increased number of contacts to the study team. Once the study participants were accustomed to the new lancets, we received feedback they worked better. It is important when changes are made to study procedures, no matter how trivial, that the reasons for the changes are explained, and appropriate instructions included.
Real time, non-judgmental, and supportive communication was essential to the success of this study. As anticipated, more of the older cohort preferred telephone and the younger cohort email as the method of choice to communicate with the study team. When designing future decentralized studies, investigators need to be prepared for the amount of communication that may be needed to help participants navigate through research procedures. We provided constant and consistent communication with responses typically immediate or within a few hours which led to high satisfaction. There were times when the study team had difficulty in responding to the burden of inquiries. Nonetheless, we feel that this was important to maintaining the engagement of the participants and budgeting of time and personnel needs to be considered for this type of trial. Email reminders throughout the study procedures via the study portal was also a positive feature reported by our participants in the survey.
As a decentralized clinical trial during the Covid-19 pandemic, the study website was being updated on an ongoing basis, with modifications made based on the availability and changing guidelines for additional vaccine doses, and the subsequent posting of participant antibody response results. In order for study participants to view these updates, they had to refresh and/ or clear their cache each time. Some participants had difficulty with this and contacted the study team to walk them through the process. The study was designed to be totally decentralized but this approach may be limiting for studies where physical exams or additional blood tests are required (although these challenges can be overcome with adequate resources) [29,30]. It does demonstrate how many study activities can be completed remotely and were acceptable to participants. As we did not have a comparative arm wherein all procedures and questionnaires were performed at a study site, and as many of our participants had never previously participated in research, we cannot conclude which method would be preferred. Given the need to participate through an electronic device, our study could not be generalizable to those who were unable to use such devices or who could not communicate in English and thus is not generalizable to the Ontario population receiving Covid-19 vaccines. The decentralized study did enable us to recruit a large number of participants in a short period time from around the province [31]. and enabled the participation of those who could not attend research facilities due to travel or mobility restrictions. We believe this to how aided in recruitment of a large number of elderly persons who might otherwise not be able to participate [32,33]. As we depended on self -report, there is no source documentation as is a feature of traditional clinical trials.

Conclusion
STOPCoV was designed in the midst of the global Covid-19 pandemic. The study was virtually accessible for both younger and older participants and satisfaction with the study procedures and communication was high. The ease of remote access allowed study activities to be performed during Covid-19 restrictions and enabled participation of those living remotely from a study center. Despite digital challenges, study retention remains high. Availability of study staff to assist with procedures and to answer questions was essential. Instructional videos to guide self-sampling were valued. Research ethics boards are acknowledging that consent and study procedures can be done remotely.